__nbid__ = '0038'
__author__ = 'Susan Ridgway <susan.ridgway@noirlab.edu>, Vinicius Placco <vinicius.placco@noirlab.edu>'
__version__ = '20240606' # yyyymmdd
__datasets__ = ['GS-2018A-Q-207'] # Data comes from the Gemini archive, GS-2018A-Q-207
__keywords__ = ['DRAGONS', 'Gemini', 'GMOS', 'data reduction']
Gemini GMOS photometry data reduction using DRAGONS Python API¶
Authors: Susan Ridgway, Vinicius Placco
Adapted from https://dragons.readthedocs.io/projects/gmosimg-drtutorial/en/v3.1.0/
Table of contents¶
- Goals
- Summary
- Disclaimers and attribution
- Imports and setup
- Downloading data for reduction
- Set up calibration service
- Collect bias files, flat files, and science frames
- Create combined flat and bias frames
- Create the reduced, combined final science frame
- Display stacked final image
- Clean-up (optional)
- Resources
Goals¶
Showcase how to perform GMOS imaging data reduction using the Gemini DRAGONS package on the Data Lab science platform. Uses a custom DRAGONS kernel "DRAGONS (Py3.7)". The steps include downloading data from the Gemini archive, setting up a DRAGONS calibration service, processing of flats, bias, and science frames, and finally the creation of a single combined stacked image.
Summary¶
DRAGONS is a Python-based astronomical data reduction platform written by the Gemini Science User Support Department. It currently can be used to reduce imaging data from Gemini instruments GMOS, NIRI, Flamingos 2, GSAOI, and GNIRS, and spectroscopic data in GMOS longslit mode. Linked here is a general list of guides, manuals and tutorials about the use of DRAGONS: https://dragons.readthedocs.io/en/v3.1.0/
The DRAGONS kernel has been made available in the Data Lab environment, which should allow users to access the routines without being dependent on installing the software in their local machines.
In this notebook, we present an example of a DRAGONS Jupyter notebook that works in the Data Lab environment to fully reduce example Gemini South GMOS G band imaging data. This is a version of the DRAGONS Jupyter notebook tutorial presented here: https://gitlab.com/nsf-noirlab/csdc/usngo/DRAGONS_tutorials/-/blob/main/GMOS_IM_FIELD.ipynb
This notebook will not present all of the details of the many options available to adjust or optimize the DRAGONS GMOS data reduction process, rather will just show one example of a standard reduction of a GMOS imaging dataset. More extensive explanations can be found in the general DRAGONS GMOS data reduction tutorial from Gemini linked here: https://dragons.readthedocs.io/projects/gmosimg-drtutorial/en/v3.1.0/
The data used in this notebook example is GMOS G band imaging from the Gemini archive of the galaxy NGC 5018 from the Gemini South program "The Evolutionary History of NGC 5018", PI: L. Sesto, program ID GS-2018A-Q-207. More program information is given here: https://archive.gemini.edu/programinfo/GS-2018A-Q-207. The final reduced science image combines 5 science frames of 460 seconds each, that were dithered between each exposure.
Disclaimer & attribution¶
Disclaimers¶
Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.
Acknowledgments¶
If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:
This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.
If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:
This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.
In either case please cite the following papers:
Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445
Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411
If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:
- Juneau et al., "Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097
If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}
And if you are using SPARCL, please also add \software{SPARCL} and cite:
- Juneau et al., "SPARCL: SPectra Analysis and Retrievable Catalog Lab", Conference Proceedings for ADASS XXXIII, 2024 https://doi.org/10.48550/arXiv.2401.05576
The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.
For this notebook specifically, please acknowledge:
DRAGONS publication: Labrie et al., "DRAGONS - Data Reduction for Astronomy from Gemini Observatory North and South", ASPC, 523, 321L, https://ui.adsabs.harvard.edu/abs/2019ASPC..523..321L/abstract
DRAGONS open source software publication: https://zenodo.org/record/7776065#.ZDg5qOzMLUI
Imports and setup¶
# std lib
import glob
# 3rd party
from astropy.io import fits
from astropy.wcs import WCS
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
%matplotlib inline
# DRAGONS
from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce
from gempy.adlibrary import dataselect
from gempy.utils import logutils
# Setup a logfile to keep track of the DRAGONS reduction
logutils.config(file_name='reduce.log')
Downloading the data from the Gemini archive¶
Create a list of the file names of the raw GMOS G band imaging data, and use wget to download these from the Gemini archive.
%%bash
# create file that lists FITS files to be downloaded
echo "\
http://archive.gemini.edu/file/S20180419S0098.fits
http://archive.gemini.edu/file/S20180419S0099.fits
http://archive.gemini.edu/file/S20180419S0100.fits
http://archive.gemini.edu/file/S20180419S0101.fits
http://archive.gemini.edu/file/S20180419S0102.fits
http://archive.gemini.edu/file/S20180419S0207.fits
http://archive.gemini.edu/file/S20180419S0208.fits
http://archive.gemini.edu/file/S20180419S0209.fits
http://archive.gemini.edu/file/S20180419S0210.fits
http://archive.gemini.edu/file/S20180419S0211.fits
http://archive.gemini.edu/file/S20180423S0050.fits
http://archive.gemini.edu/file/S20180423S0051.fits
http://archive.gemini.edu/file/S20180423S0052.fits
http://archive.gemini.edu/file/S20180423S0053.fits
http://archive.gemini.edu/file/S20180423S0054.fits\
" > gmos.list
# download with wget FITS files from Gemini archive for processing (in quiet mode)
wget --no-check-certificate -N -q -i gmos.list
Set up the Gemini DRAGONS calibration service¶
Starts the calibration service for DRAGONS. The w flag removes all previous calibration databases that might have been set up.
caldb = cal_service.set_local_database()
caldb.init("w")
Collect bias files, flat files, and science frames¶
This cell stores the names of the files and prints a sorted list for visual inspection.
all_files = glob.glob('S20*[0-9].fits')
all_files.sort()
print(all_files)
['S20180419S0098.fits', 'S20180419S0099.fits', 'S20180419S0100.fits', 'S20180419S0101.fits', 'S20180419S0102.fits', 'S20180419S0207.fits', 'S20180419S0208.fits', 'S20180419S0209.fits', 'S20180419S0210.fits', 'S20180419S0211.fits', 'S20180423S0050.fits', 'S20180423S0051.fits', 'S20180423S0052.fits', 'S20180423S0053.fits', 'S20180423S0054.fits']
Here the raw data is sorted into types (bias, flats, and science images) using dataselect and lists are made of the raw frames of each type. These lists are used as input to the later combination steps.
list_biases = dataselect.select_data(all_files,['BIAS'],[])
list_flats = dataselect.select_data(all_files,['FLAT'],[],
dataselect.expr_parser('filter_name=="g"'))
list_science = dataselect.select_data(all_files,[],['CAL'],
dataselect.expr_parser('(observation_class=="science" and filter_name=="g")'))
Create combined bias frames and flat frames¶
The bias frames are reduced and stacked to create a combined bias frame. The flat frames are reduced, bias corrected and stacked to create a combined flat frame. After each reduction, the output is added to the calibration database.
reduce_bias = Reduce()
reduce_bias.files.extend(list_biases)
reduce_bias.runr()
caldb.add_cal(reduce_bias.output_filenames[0])
reduce_flats = Reduce()
reduce_flats.files.extend(list_flats)
reduce_flats.runr()
caldb.add_cal(reduce_flats.output_filenames[0])
All submitted files appear valid: S20180423S0050.fits ... S20180423S0054.fits, 5 files submitted.
Ginga not installed, use other viewer, or no viewer
Produce the final stacked, reduced science image¶
In this step, the standard DRAGONS GMOS data reduction protocol is followed: first, the science frames are processed, and a rough combined frame is created that is then used to further flat field and cosmic ray reject the input science frames. These further processed, dithered input frames are then re-combined to maximize signal-to-noise and produce a final combined science frame. Note: This process is iterative: 1st, detects sources in individual frames, cross correlated to see which are cross correlated (not CRs) and also uses these to derive offsets, and CR correct. A combined (median, no rejection) sky frame is made, which is then subtracted from each image the final science frame is created after CR Rejecting and sky subtracting the input files, using combination = mean and rejection sigclip
reduce_science = Reduce()
reduce_science.files.extend(list_science)
reduce_science.runr()
Display the fully reduced image¶
Read the reduced spectrum with astropy and print information about the various FITS extensions provided by the DRAGONS data reduction. Also read the WCS (World Coordinate System) for plotting purposes.
image_file = "S20180419S0098_image.fits"
hdu_list = fits.open(image_file)
wcs = WCS(hdu_list[1].header)
hdu_list.info()
WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATE-OBS to '2018-04-19T03:27:58.541' from MJD-OBS'. [astropy.wcs.wcs] FITSFixedWarning: 'datfix' made the change 'Set DATE-OBS to '2018-04-19T03:27:58.541' from MJD-OBS'.
Filename: S20180419S0098_image.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 250 () 1 SCI 1 ImageHDU 148 (3245, 2182) float32 2 VAR 1 ImageHDU 148 (3245, 2182) float32 3 DQ 1 ImageHDU 148 (3245, 2182) int16 (rescales to uint16) 4 PROVENANCE 1 BinTableHDU 17 7R x 4C [28A, 128A, 128A, 128A] 5 PROVHISTORY 1 BinTableHDU 17 18R x 4C [128A, 288A, 28A, 28A]
Get (x,y) data from the SCI extension and print the array's shape
image_data = fits.getdata(image_file, ext=1)
print(image_data.shape)
(2182, 3245)
Display the image using standard matplotlib.pyplot routines. Note the use of the WCS for the coordinates. The norm=LogNorm function is used to improve the contrast in the image.
plt.figure(figsize = (15,15))
plt.subplot(projection=wcs)
plt.imshow(image_data,cmap='gray',norm=LogNorm(vmin=3000, vmax=60000))
plt.xlabel('Right Ascension [degree]',fontsize=14,fontweight='bold')
plt.ylabel('Declination [degree]',fontsize=14,fontweight='bold')
plt.xlim(900,2500)
plt.ylim(1600,600)
plt.show()
Clean-up (optional)¶
Erase all fits files, lists, logs, and calibrations created during the reduction. For that, uncomment all lines in the cell below and execute.
# %%bash
# rm -f *S2018*.fits gmos.list reduce.log # remove files
# rm -rf calibrations/ # remove directories
Resources¶
US NGO DRAGONS tutorials: https://gitlab.com/nsf-noirlab/csdc/usngo/DRAGONS_tutorials/
US NGO DRAGONS tutorial on which this example is based: https://gitlab.com/nsf-noirlab/csdc/usngo/DRAGONS_tutorials/-/blob/main/GMOS_IM_FIELD.ipynb
Gemini DRAGONS list of manuals and tutorials: https://dragons.readthedocs.io/en/v3.1.0/
Gemini DRAGONS GMOS imaging data reduction tutorial: https://dragons.readthedocs.io/projects/gmosimg-drtutorial/en/v3.1.0/
Gemini archival link to the program from which these data were obtained: https://archive.gemini.edu/programinfo/GS-2018A-Q-207
General DataLab/Jupyter Notebook/python Links: